On the Support Vector Machine

Classification is a fundamental problem at the intersection of machine learning and statistics. Machine learning methods have enjoyed considerable empirical success. However, they often have an ad hoc quality. It is desirable to have hard theoretical results which might highlight specific quantitative advantages of these methods. The statistical methods often tackle the classification problem through density estimation or regression. Theoretical properties of these statistical methods can be established, but only under the assumption of a fixed order of smoothness. Whether these methods work well when the assumptions are violated is not clear. The support vector machine (SVM) methodology is a rapidly growing area in machine learning, and is receiving considerable attention in recent years. The SVM has proved highly successful in a number of practical classification studies. In this paper we show that the SVM enjoys excellent theoretical properties which explain the good performance of the SVM. We show that the SVM approaches the the theoretical optimal classification rule (the Bayes rule) in a direct fashion, and its expected misclassification rate quickly converges to that of the Bayes rule. The results are established under very general conditions allowing discontinuity. They testify to the fact that classification is easier than density estimation and regression, and show that the SVM works by taking advantage of this. The results pinpoint the exact mechanism behind the SVM, and clarify the advantage and limitation of the SVM, thus give insights on how the SVM can be extended systematically.